{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4b4a0d60",
   "metadata": {},
   "source": [
    "# VectorStore-backed Retriever\n",
    "\n",
    "- Author: [Erika Park](https://www.linkedin.com/in/yeonseo-park-094193198/)\n",
    "- Designer: [Erika Park](https://www.linkedin.com/in/yeonseo-park-094193198/)\n",
    "- Proofread : [jishin86](https://github.com/jishin86)\n",
    "- This is a part of [LangChain Open Tutorial](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial)\n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/10-Retriever/01-VectorStoreRetriever.ipynb)\n",
    "[![Open in GitHub](https://img.shields.io/badge/Open%20in%20GitHub-181717?style=flat-square&logo=github&logoColor=white)](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/10-Retriever/01-VectorStoreRetriever.ipynb)\n",
    "\n",
    "## Overview\n",
    "This tutorial provides a comprehensive guide to building and optimizing a **VectorStore-backed retriever** using LangChain. It covers the foundational steps of creating a vector store with FAISS(Facebook AI Similarity Search) and explores advanced retrieval strategies for improving search accuracy and efficiency.\n",
    "\n",
    "A **VectorStore-backed retriever** is a document retrieval system that leverages a vector store to search for documents based on their vector representations. This approach enables efficient similarity-based search for handling unstructured data.\n",
    "\n",
    "\n",
    "### RAG (Retrieval-Augmented Generation) Workflow\n",
    "![rag-flow.png](./assets/01-vectorstore-retriever-rag-flow.png)\n",
    "\n",
    "The diagram above illustrates the  **document search and response generation** workflow within a RAG system. \n",
    "\n",
    "The steps include:\n",
    "\n",
    "1. Document Loading: Importing raw documents.  \n",
    "2. Text Chunking: Splitting text into manageable chunks.  \n",
    "3. Vector Embedding: Converting the text into numerical vectors using an embedding model.  \n",
    "4. Store in Vector Database: Storing the generated embeddings in a vector database for efficient retrieval.\n",
    "\n",
    "During the query phase:\n",
    "- Steps: User Query → Embedding → Search in VectorStore → Relevant Chunks Retrieved → LLM Generates Response\n",
    "- The user's query is transformed into an embedding vector using an embedding model.\n",
    "- This query embedding is compared against stored document vectors within the vector database to **retrieve the most relevant results**.\n",
    "- The retrieved chunks are passed to a Large Language Model (LLM), which generates a final response based on the retrieved information.\n",
    "\n",
    "This tutorial aims to explore and optimize the VectorStore → Relevant Chunks Retrieved → LLM Generates Response stages. It will cover advanced retrieval techniques to improve the accuracy and relevance of the responses.\n",
    "\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "- [Overview](#overview)\n",
    "- [Environment Setup](#environment-setup)\n",
    "- [Initializing and Using VectorStoreRetriever](#initializing-and-using-vectorstoreretriever)\n",
    "- [Dynamic Configuration (Using ConfigurableField)](#dynamic-configuration-using-configurablefield)\n",
    "- [Using Separate Query & Passage Embedding Models](#using-separate-query--passage-embedding-models)\n",
    "\n",
    "### References\n",
    "\n",
    "- [How to use a vectorstore as a retriever](https://python.langchain.com/docs/how_to/vectorstore_retriever/)\n",
    "- [Maximum Marginal Relevance (MMR)](https://community.fullstackretrieval.com/retrieval-methods/maximum-marginal-relevance)\n",
    "- [Upstage-Embeddings](https://console.upstage.ai/docs/capabilities/embeddings)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c6af75",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "Set up the environment. You may refer to [Environment Setup](https://wikidocs.net/257836) for more details.\n",
    "\n",
    "**[Note]**\n",
    "- ```langchain-opentutorial``` is a package that provides a set of easy-to-use environment setup, useful functions, and utilities for tutorials. \n",
    "- You can checkout out the [```langchain-opentutorial```](https://github.com/LangChain-OpenTutorial/langchain-opentutorial-pypi) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a9a9e966",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install langchain-opentutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e3c34330",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "from langchain_opentutorial import package\n",
    "\n",
    "package.install(\n",
    "    [\n",
    "        \"langchain_opentutorial\",\n",
    "        \"langchain_openai\",\n",
    "        \"langchain_community\",\n",
    "        \"langchain_text_splitters\",\n",
    "        \"langchain_core\",\n",
    "        \"langchain_upstage\",\n",
    "        \"faiss-cpu\"\n",
    "    ],\n",
    "    verbose=False,\n",
    "    upgrade=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c2bf9b80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environment variables have been set successfully.\n"
     ]
    }
   ],
   "source": [
    "# Set environment variables\n",
    "from langchain_opentutorial import set_env\n",
    "\n",
    "set_env(\n",
    "    {\n",
    "        # \"OPENAI_API_KEY\": \"\",\n",
    "        # \"LANGCHAIN_API_KEY\": \"\",\n",
    "        # \"UPSTAGE_API_KEY\": \"\",\n",
    "        \"LANGCHAIN_TRACING_V2\": \"true\",\n",
    "        \"LANGCHAIN_ENDPOINT\": \"https://api.smith.langchain.com\",\n",
    "        \"LANGCHAIN_PROJECT\": \"VectorStore Retriever\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90c24f41",
   "metadata": {},
   "source": [
    "You can alternatively set API keys such as ```OPENAI_API_KEY``` in a ```.env``` file and load them.\n",
    "\n",
    "[Note] This is not necessary if you've already set the required API keys in previous steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cfaffe52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Configuration file to manage the API KEY as an environment variable\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Load API KEY information\n",
    "load_dotenv(override=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e66b00",
   "metadata": {},
   "source": [
    "## Initializing and Using VectorStoreRetriever\n",
    "\n",
    "This section demonstrates how to load documents using OpenAI embeddings and create a vector database using FAISS.\n",
    "\n",
    "- The example below showcases how to use OpenAI embeddings for document loading and FAISS for vector database creation.\n",
    "- Once the vector database is created, it can be loaded and queried using retrieval methods such as **Similarity Search** and **Maximal Marginal Relevance (MMR)** to search for relevant text within the vector store."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dc586a1",
   "metadata": {},
   "source": [
    "📌 **Creating a Vector Store (Using FAISS)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ee9b98ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 351, which is longer than the specified 300\n",
      "Created a chunk of size 343, which is longer than the specified 300\n",
      "Created a chunk of size 307, which is longer than the specified 300\n",
      "Created a chunk of size 316, which is longer than the specified 300\n",
      "Created a chunk of size 341, which is longer than the specified 300\n",
      "Created a chunk of size 321, which is longer than the specified 300\n",
      "Created a chunk of size 303, which is longer than the specified 300\n",
      "Created a chunk of size 325, which is longer than the specified 300\n",
      "Created a chunk of size 315, which is longer than the specified 300\n",
      "Created a chunk of size 304, which is longer than the specified 300\n",
      "Created a chunk of size 385, which is longer than the specified 300\n",
      "Created a chunk of size 349, which is longer than the specified 300\n",
      "Created a chunk of size 376, which is longer than the specified 300\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.vectorstores import FAISS\n",
    "from langchain_openai.embeddings import OpenAIEmbeddings\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "\n",
    "# Load the file using TextLoader\n",
    "loader = TextLoader(\"./data/01-vectorstore-retriever-appendix-keywords.txt\", encoding=\"utf-8\")\n",
    "documents = loader.load()\n",
    "\n",
    "# split the text into chunks\n",
    "text_splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0)\n",
    "split_docs = text_splitter.split_documents(documents) # Split into smaller chunks\n",
    "\n",
    "# Initialize the OpenAI embedding model\n",
    "embeddings = OpenAIEmbeddings()\n",
    "\n",
    "# Create a FAISS vector database\n",
    "db = FAISS.from_documents(split_docs, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fbd962",
   "metadata": {},
   "source": [
    "📌 **1. Initializing and Using VectorStoreRetriever (```as_retriever``` )**\n",
    "\n",
    "The ```as_retriever``` method allows you to convert a vector database into a retriever, enabling efficient document search and retrieval from the vector store.\n",
    "\n",
    "**How It Works**:\n",
    "* The ```as_retriever()``` method transforms a vector store (like FAISS) into a retriever object, making it compatible with LangChain's retrieval workflows.\n",
    "* This retriever can then be directly used with RAG pipelines or combined with Large Language Models (LLMs) for building intelligent search systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "05d21318",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic Retriever Creation (Similarity Search)\n",
    "retriever = db.as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d51d7bc",
   "metadata": {},
   "source": [
    "**Advanced Retriever Configuration**\n",
    "\n",
    "The ```as_retriever``` method allows you to configure advanced retrieval strategies, such as **similarity search**, **MMR (Maximal Marginal Relevance)**, and **similarity score threshold-based filtering**.\n",
    "\n",
    "\n",
    "**Parameters:**\n",
    "\n",
    "- ```**kwargs``` : Keyword arguments passed to the retrieval function:\n",
    "   - ```search_type``` : Specifies the search method.\n",
    "     - ```\"similarity\"``` : Returns the most relevant documents based on cosine similarity.\n",
    "     - ```\"mmr\"``` : Utilizes the Maximal Marginal Relevance algorithm, balancing **relevance** and **diversity**.\n",
    "     - ```\"similarity_score_threshold\"``` : Returns documents with a similarity score above a specified threshold.\n",
    "   - ```search_kwargs``` : Additional search options for fine-tuning results:\n",
    "     - ```k``` : Number of documents to return (default: ```4``` ).\n",
    "     - ```score_threshold``` : Minimum similarity score for the ```\"similarity_score_threshold\"``` search type (e.g., ```0.8``` ).\n",
    "     - ```fetch_k``` : Number of documents initially retrieved during an MMR search (default: ```20``` ).\n",
    "     - ```lambda_mult``` : Controls diversity in MMR results (```0``` = maximum diversity, ```1``` = maximum relevance, default: ```0.5``` ).\n",
    "     - ```filter``` : Metadata filtering for selective document retrieval.\n",
    "\n",
    "\n",
    " **Return Value:**\n",
    "\n",
    "- ```VectorStoreRetriever```: An initialized retriever object that can be directly queried for document search tasks.\n",
    "\n",
    "\n",
    "**Notes:**\n",
    "- Supports multiple search strategies (```similarity``` , ```MMR``` , ```similarity_score_threshold``` ).\n",
    "- MMR improves result diversity while preserving relevance by reducing redundancy in results.\n",
    "- Metadata filtering enables selective document retrieval based on document properties.\n",
    "- The ```tags``` parameter can be used to label retrievers for better organization and easier identification.\n",
    "\n",
    " **Cautions:**\n",
    "- Diversity Control with MMR:\n",
    "  - Adjust both ```fetch_k``` (number of documents initially retrieved) and ```lambda_mult``` (diversity control factor) carefully for optimal balance.\n",
    "  - ```lambda_mult```\n",
    "    - Lower values (< 0.5) → Prioritize diversity.\n",
    "    - Higher values (> 0.5) → Prioritize relevance.\n",
    "  - set ```fetch_k``` higher than ```k``` for effective diversity control.\n",
    "- Threshold Settings: \n",
    "  - Using a high ```score_threshold``` (e.g., 0.95) can lead to zero results.\n",
    "- Metadata Filtering: \n",
    "  - Ensure the metadata structure is well-defined before applying filters.\n",
    "- Balanced Configuration:\n",
    "  - Maintain a proper balance between ```search_type``` and ```search_kwargs``` settings for optimal retrieval performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c038da2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Semantic Search\n",
      "VectorStore\n",
      "\n",
      "Definition: A vector store is a system for storing data in vector format, often used for search, classification, and data analysis tasks.\n",
      "Example: Storing word embeddings in a database for fast retrieval of similar words.\n",
      "Related Keywords: Embedding, Database, Vectorization\n",
      "Definition: Semantic search is a method of retrieving results based on the meaning of the user's query, going beyond simple keyword matching.\n",
      "Example: If a user searches for \"solar system planets,\" the search returns information about related planets like Jupiter and Mars.\n",
      "Related Keywords: Natural Language Processing, Search Algorithms, Data Mining\n",
      "Definition: Keyword search is the process of finding information based on specific keywords entered by the user. It is commonly used in search engines and database systems as a fundamental search method.\n",
      "Example: If a user searches for \"coffee shop in Seoul,\" the search engine returns a list of related coffee shops.\n",
      "Related Keywords: Search Engine, Data Retrieval, Information Search\n",
      "Definition: FAISS is a high-speed similarity search library developed by Facebook, designed for efficient vector searches in large datasets.\n",
      "Example: Searching for similar images in a dataset of millions using FAISS.\n",
      "Related Keywords: Vector Search, Machine Learning, Database Optimization\n"
     ]
    }
   ],
   "source": [
    "retriever = db.as_retriever(\n",
    "    search_type=\"similarity_score_threshold\", \n",
    "    search_kwargs={\n",
    "        \"k\": 5,  # Return the top 5 most relevant documents\n",
    "        \"score_threshold\": 0.7  # Only return documents with a similarity score of 0.7 or higher\n",
    "    }\n",
    ")\n",
    "# Perform the search\n",
    "query = \"Explain the concept of vector search.\"\n",
    "results = retriever.invoke(query)\n",
    "\n",
    "# Display search results\n",
    "for doc in results:\n",
    "    print(doc.page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d741849",
   "metadata": {},
   "source": [
    "### Retriever's ```invoke()``` Method\n",
    "\n",
    "The ```invoke()``` method is the primary entry point for interacting with a Retriever. It is used to search and retrieve relevant documents based on a given query.\n",
    "\n",
    "**How It Works** :\n",
    "1. Query Submission: A user query is provided as input.\n",
    "2. Embedding Generation: The query is converted into a vector representation (if necessary).\n",
    "3. Search Process: The retriever searches the vector database using the specified search strategy (similarity, MMR, etc.).\n",
    "4. Results Return: The method returns a list of relevant document chunks.\n",
    "\n",
    " **Parameters:**\n",
    "- ```input``` (Required):\n",
    "   - The query string provided by the user.\n",
    "   - The query is converted into a vector and compared with stored document vectors for similarity-based retrieval.\n",
    "\n",
    "- ```config``` (Optional):\n",
    "   - Allows for fine-grained control over the retrieval process.\n",
    "   - Can be used to specify **tags, metadata insertion, and search strategies**.\n",
    "\n",
    "- ```**kwargs``` (Optional):\n",
    "   - Enables direct passing of ```search_kwargs``` for advanced configuration.\n",
    "   - Example options include:\n",
    "     - ```k``` : Number of documents to return.\n",
    "     - ```score_threshold``` : Minimum similarity score for a document to be included.\n",
    "     - ```fetch_k``` : Number of documents initially retrieved in MMR searches.\n",
    "\n",
    "\n",
    " **Return Value:**\n",
    "- ```List[Document]```:\n",
    "   - Returns a list of document objects containing the retrieved text and metadata.\n",
    "   - Each document object includes:\n",
    "     - ```page_content``` : The main content of the document.\n",
    "     - ```metadata``` : Associated metadata with the document (e.g., source, tags).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51834bb6",
   "metadata": {},
   "source": [
    "**Usage Example 1: Basic Usage (Synchronous Search)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "257d2ade",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embedding\n",
      "=========================================================\n",
      "Definition: Embedding is the process of converting text data, such as words or sentences, into continuous low-dimensional vectors. This allows computers to understand and process text.\n",
      "Example: The word \"apple\" can be represented as a vector like [0.65, -0.23, 0.17].\n",
      "Related Keywords: Natural Language Processing, Vectorization, Deep Learning\n",
      "=========================================================\n",
      "Semantic Search\n",
      "=========================================================\n",
      "Deep Learning\n",
      "=========================================================\n"
     ]
    }
   ],
   "source": [
    "docs = retriever.invoke(\"What is an embedding?\")\n",
    "\n",
    "for doc in docs:\n",
    "    print(doc.page_content)\n",
    "    print(\"=========================================================\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5d5ba5",
   "metadata": {},
   "source": [
    "**Usage Example 2: Search with Options** ( ```search_kwargs``` )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "643a6491",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VectorStore\n",
      "\n",
      "Definition: A vector store is a system for storing data in vector format, often used for search, classification, and data analysis tasks.\n",
      "Example: Storing word embeddings in a database for fast retrieval of similar words.\n",
      "Related Keywords: Embedding, Database, Vectorization\n",
      "=========================================================\n"
     ]
    }
   ],
   "source": [
    "# search options: top 5 results with a similarity score ≥ 0.7\n",
    "docs = retriever.invoke(\n",
    "    \"What is a vector database?\",\n",
    "    search_kwargs={\"k\": 5, \"score_threshold\": 0.7}\n",
    ")\n",
    "for doc in docs:\n",
    "    print(doc.page_content)\n",
    "    print(\"=========================================================\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e360a59",
   "metadata": {},
   "source": [
    "**Usage Example 3: Using** ```config``` **and** ```**kwargs``` **(Advanced Configuration)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d58154d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔍 [Search Result 1]\n",
      "📄 Document Content: Definition: A DataFrame is a tabular data structure with rows and columns, commonly used for data analysis and manipulation.\n",
      "Example: Pandas DataFrame can store data like an Excel sheet and perform operations like filtering and grouping.\n",
      "Related Keywords: Data Analysis, Pandas, Data Manipulation\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "\n",
      "🔍 [Search Result 2]\n",
      "📄 Document Content: Schema\n",
      "\n",
      "Definition: A schema defines the structure of a database or file, describing how data is stored and organized.\n",
      "Example: A database schema can specify table columns, data types, and constraints.\n",
      "Related Keywords: Database, Data Modeling, Data Management\n",
      "\n",
      "DataFrame\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "\n",
      "🔍 [Search Result 3]\n",
      "📄 Document Content: Pandas\n",
      "\n",
      "Definition: Pandas is a Python library for data analysis and manipulation, offering tools for working with structured data.\n",
      "Example: Pandas can read CSV files, clean data, and perform statistical analysis.\n",
      "Related Keywords: Data Analysis, Python, Data Manipulation\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "\n",
      "🔍 [Search Result 4]\n",
      "📄 Document Content: Data Mining\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.runnables.config import RunnableConfig\n",
    "\n",
    "# Create a RunnableConfig with tags and metadata\n",
    "config = RunnableConfig(\n",
    "    tags=[\"retrieval\", \"faq\"],  ## Adding tags for query categorization\n",
    "    metadata={\"project\": \"vectorstore-tutorial\"}  # Project-specific metadata for traceability\n",
    ")\n",
    "# Perform a query using advanced configuration settings\n",
    "docs = retriever.invoke(\n",
    "    input=\"What is a DataFrame?\", \n",
    "    config=config,  # Applying the config with tags and metadata\n",
    "    search_kwargs={\n",
    "        \"k\": 3,                   \n",
    "        \"score_threshold\": 0.8   \n",
    "    }\n",
    ")\n",
    "#  Display the search results\n",
    "for idx, doc in enumerate(docs):\n",
    "    print(f\"\\n🔍 [Search Result {idx + 1}]\")\n",
    "    print(\"📄 Document Content:\", doc.page_content)\n",
    "    print(\"🗂️ Metadata:\", doc.metadata)\n",
    "    print(\"=\" * 60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab94c2c",
   "metadata": {},
   "source": [
    "### Max Marginal Relevance (MMR)\n",
    "\n",
    "The **Maximal Marginal Relevance (MMR)** search method is a document retrieval algorithm designed to reduce redundancy by balancing relevance and diversity when returning results.\n",
    "\n",
    "**How MMR Works:**\n",
    "Unlike basic similarity-based searches that return the most relevant documents based solely on similarity scores, MMR considers two critical factors:\n",
    "1. Relevance: Measures how closely the document matches the user's query.\n",
    "2. Diversity: Ensures the retrieved documents are distinct from each other to avoid repetitive results.\n",
    "\n",
    " **Key Parameters:**\n",
    "- ```search_type=\"mmr\"```: Activates the MMR retrieval strategy.  \n",
    "- ```k```: The number of documents returned after applying diversity filtering(default: ```4```).  \n",
    "- ```fetch_k```: Number of documents initially retrieved before applying diversity filtering (default: ```20```).  \n",
    "- ```lambda_mult```: Diversity control factor (```0 = max diversity``` , ```1 = max relevance``` , default: ```0.5```)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8144a926",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Query]: What is an embedding?\n",
      "\n",
      "📄 [Document 1]\n",
      "📖 Document Content: Embedding\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 2]\n",
      "📖 Document Content: Definition: Embedding is the process of converting text data, such as words or sentences, into continuous low-dimensional vectors. This allows computers to understand and process text.\n",
      "Example: The word \"apple\" can be represented as a vector like [0.65, -0.23, 0.17].\n",
      "Related Keywords: Natural Language Processing, Vectorization, Deep Learning\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 3]\n",
      "📖 Document Content: TF-IDF (Term Frequency-Inverse Document Frequency)\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# MMR Retriever Configuration (Balancing Relevance and Diversity)\n",
    "retriever = db.as_retriever(\n",
    "    search_type=\"mmr\", \n",
    "    search_kwargs={\n",
    "        \"k\": 3,                \n",
    "        \"fetch_k\": 10,           \n",
    "        \"lambda_mult\": 0.6  # Balancing Similarity and Diversity (0.6: Slight Emphasis on Diversity)\n",
    "    }\n",
    ")\n",
    "\n",
    "query = \"What is an embedding?\"\n",
    "docs = retriever.invoke(query)\n",
    "\n",
    "#  Display the search results\n",
    "print(f\"\\n🔎 [Query]: {query}\\n\")\n",
    "for idx, doc in enumerate(docs):\n",
    "    print(f\"📄 [Document {idx + 1}]\")\n",
    "    print(\"📖 Document Content:\", doc.page_content)\n",
    "    print(\"🗂️ Metadata:\", doc.metadata)\n",
    "    print(\"=\" * 60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "902babfe",
   "metadata": {},
   "source": [
    "### Similarity Score Threshold Search\n",
    "\n",
    "**Similarity Score Threshold Search** is a retrieval method where only documents exceeding a predefined similarity score are returned. This approach helps filter out low-relevance results, ensuring that the returned documents are highly relevant to the query.\n",
    "\n",
    "**Key Features:**\n",
    "- Relevance Filtering: Returns only documents with a similarity score above the specified threshold.\n",
    "- Configurable Precision: The threshold is adjustable using the ```score_threshold``` parameter.\n",
    "- Search Type Activation: Enabled by setting ```search_type=\"similarity_score_threshold\"``` .\n",
    "\n",
    "This search method is ideal for tasks requiring **highly precise** results, such as fact-checking or answering technical queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b6509f52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Query]: What is Word2Vec?\n",
      "\n",
      "📄 [Document 1]\n",
      "📖 Document Content: Word2Vec\n",
      "\n",
      "Definition: Word2Vec is a technique in NLP that maps words into a vector space, representing their semantic relationships based on context.\n",
      "Example: In Word2Vec, \"king\" and \"queen\" would be represented by vectors close to each other.\n",
      "Related Keywords: NLP, Embeddings, Semantic Similarity\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 2]\n",
      "📖 Document Content: Definition: Embedding is the process of converting text data, such as words or sentences, into continuous low-dimensional vectors. This allows computers to understand and process text.\n",
      "Example: The word \"apple\" can be represented as a vector like [0.65, -0.23, 0.17].\n",
      "Related Keywords: Natural Language Processing, Vectorization, Deep Learning\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 3]\n",
      "📖 Document Content: TF-IDF (Term Frequency-Inverse Document Frequency)\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 4]\n",
      "📖 Document Content: Tokenizer\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 5]\n",
      "📖 Document Content: Semantic Search\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# Retriever Configuration (Similarity Score Threshold Search)\n",
    "retriever = db.as_retriever(\n",
    "    search_type=\"similarity_score_threshold\",  \n",
    "    search_kwargs={\n",
    "        \"score_threshold\": 0.6,  \n",
    "        \"k\": 5                \n",
    "    }\n",
    ")\n",
    "# Execute the query\n",
    "query = \"What is Word2Vec?\"\n",
    "docs = retriever.invoke(query)\n",
    "\n",
    "# Display the search results \n",
    "print(f\"\\n🔎 [Query]: {query}\\n\")\n",
    "if docs:\n",
    "    for idx, doc in enumerate(docs):\n",
    "        print(f\"📄 [Document {idx + 1}]\")\n",
    "        print(\"📖 Document Content:\", doc.page_content)\n",
    "        print(\"🗂️ Metadata:\", doc.metadata)\n",
    "        print(\"=\" * 60)\n",
    "else:\n",
    "    print(\"⚠️ No relevant documents found. Try lowering the similarity score threshold.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b16c14f",
   "metadata": {},
   "source": [
    "### Configuring ```top_k``` (Adjusting the Number of Returned Documents)\n",
    "\n",
    "- The parameter ```k``` specifies the number of documents returned during a vector search. It determines how many of the **top-ranked** documents (based on similarity score) will be retrieved from the vector database.\n",
    "\n",
    "- The number of documents retrieved can be adjusted by setting the ```k``` value within the ```search_kwargs```.  \n",
    "- For example, setting ```k=1``` will return only the **top 1 most relevant document** based on similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "081e0134",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Query]: What is an embedding?\n",
      "\n",
      "📄 [Document 1]\n",
      "📖 Document Content: Embedding\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# Retriever Configuration (Return Only the Top 1 Document)\n",
    "retriever = db.as_retriever(\n",
    "    search_kwargs={\n",
    "        \"k\": 1  # Return only the top 1 most relevant document\n",
    "    }\n",
    ")\n",
    "\n",
    "query = \"What is an embedding?\"\n",
    "docs = retriever.invoke(query)\n",
    "\n",
    "#  Display the search results \n",
    "print(f\"\\n🔎 [Query]: {query}\\n\")\n",
    "if docs:\n",
    "    for idx, doc in enumerate(docs):\n",
    "        print(f\"📄 [Document {idx + 1}]\")\n",
    "        print(\"📖 Document Content:\", doc.page_content)\n",
    "        print(\"🗂️ Metadata:\", doc.metadata)\n",
    "        print(\"=\" * 60)\n",
    "else:\n",
    "    print(\"⚠️ No relevant documents found. Try increasing the `k` value.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1168ca53",
   "metadata": {},
   "source": [
    "## Dynamic Configuration (Using ```ConfigurableField``` )\n",
    "\n",
    "The ```ConfigurableField``` feature in LangChain allows for **dynamic adjustment** of search configurations, providing flexibility during query execution.\n",
    "\n",
    "**Key Features:**\n",
    "- Runtime Search Configuration: Adjust search settings without modifying the core retriever setup.\n",
    "- Enhanced Traceability: Assign unique identifiers, names, and descriptions to each parameter for improved readability and debugging.\n",
    "- Flexible Control with ```config```: Search configurations can be passed dynamically using the ```config``` parameter as a dictionary.\n",
    "\n",
    "\n",
    "**Use Cases:**\n",
    "- Switching Search Strategies: Dynamically adjust the search type (e.g., ```\"similarity\"```, ```\"mmr\"``` ).\n",
    "- Real-Time Parameter Adjustments: Modify search parameters like ```k``` , ```score_threshold``` , and ```fetch_k``` during query execution.\n",
    "- Experimentation: Easily test different search strategies and parameter combinations without rewriting code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bc25029f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import ConfigurableField \n",
    "\n",
    "# Retriever Configuration Using ConfigurableField\n",
    "retriever = db.as_retriever(search_kwargs={\"k\": 1}).configurable_fields(\n",
    "    search_type=ConfigurableField(\n",
    "        id=\"search_type\", \n",
    "        name=\"Search Type\",  # Name for the search strategy\n",
    "        description=\"The search type to use\",  # Description of the search strategy\n",
    "    ),\n",
    "    search_kwargs=ConfigurableField(\n",
    "        id=\"search_kwargs\",  \n",
    "        name=\"Search Kwargs\",  # Name for the search parameters\n",
    "        description=\"The search kwargs to use\",  # Description of the search parameters\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "721b750c",
   "metadata": {},
   "source": [
    "The following examples demonstrate how to apply dynamic search settings using ```ConfigurableField``` in LangChain.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "48f53c12",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Search Results - Basic Configuration (Top 3 Documents)]\n",
      "📄 [Document 1]\n",
      "Embedding\n",
      "============================================================\n",
      "📄 [Document 2]\n",
      "Definition: Embedding is the process of converting text data, such as words or sentences, into continuous low-dimensional vectors. This allows computers to understand and process text.\n",
      "Example: The word \"apple\" can be represented as a vector like [0.65, -0.23, 0.17].\n",
      "Related Keywords: Natural Language Processing, Vectorization, Deep Learning\n",
      "============================================================\n",
      "📄 [Document 3]\n",
      "Semantic Search\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# ✅ Search Configuration 1: Basic Search (Top 3 Documents)\n",
    "\n",
    "config_1 = {\"configurable\": {\"search_kwargs\": {\"k\": 3}}}\n",
    "\n",
    "# Execute the query\n",
    "docs = retriever.invoke(\"What is an embedding?\", config=config_1)\n",
    "\n",
    "# Display the search results\n",
    "print(\"\\n🔎 [Search Results - Basic Configuration (Top 3 Documents)]\")\n",
    "for idx, doc in enumerate(docs):\n",
    "    print(f\"📄 [Document {idx + 1}]\")\n",
    "    print(doc.page_content)\n",
    "    print(\"=\" * 60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "951851c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Search Results - Similarity Score Threshold ≥ 0.8]\n",
      "📄 [Document 1]\n",
      "Word2Vec\n",
      "\n",
      "Definition: Word2Vec is a technique in NLP that maps words into a vector space, representing their semantic relationships based on context.\n",
      "Example: In Word2Vec, \"king\" and \"queen\" would be represented by vectors close to each other.\n",
      "Related Keywords: NLP, Embeddings, Semantic Similarity\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# ✅ Search Configuration 2: Similarity Score Threshold (≥ 0.8)\n",
    "\n",
    "config_2 = {\n",
    "    \"configurable\": {\n",
    "        \"search_type\": \"similarity_score_threshold\",\n",
    "        \"search_kwargs\": {\n",
    "            \"score_threshold\": 0.8,  # Only return documents with a similarity score of 0.8 or higher\n",
    "        },\n",
    "    }\n",
    "}\n",
    "\n",
    "# Execute the query\n",
    "docs = retriever.invoke(\"What is Word2Vec?\", config=config_2)\n",
    "\n",
    "# Display the search results\n",
    "print(\"\\n🔎 [Search Results - Similarity Score Threshold ≥ 0.8]\")\n",
    "for idx, doc in enumerate(docs):\n",
    "    print(f\"📄 [Document {idx + 1}]\")\n",
    "    print(doc.page_content)\n",
    "    print(\"=\" * 60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "02e6e23d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Search Results - MMR (Diversity and Relevance Balanced)]\n",
      "📄 [Document 1]\n",
      "Word2Vec\n",
      "\n",
      "Definition: Word2Vec is a technique in NLP that maps words into a vector space, representing their semantic relationships based on context.\n",
      "Example: In Word2Vec, \"king\" and \"queen\" would be represented by vectors close to each other.\n",
      "Related Keywords: NLP, Embeddings, Semantic Similarity\n",
      "============================================================\n",
      "📄 [Document 2]\n",
      "Tokenizer\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# ✅ Search Configuration 3: MMR Search (Diversity and Relevance Balanced)\n",
    "\n",
    "config_3 = {\n",
    "    \"configurable\": {\n",
    "        \"search_type\": \"mmr\",\n",
    "        \"search_kwargs\": {\n",
    "            \"k\": 2,            # Return the top 2 most diverse and relevant documents\n",
    "            \"fetch_k\": 10,     # Initially fetch the top 10 documents before filtering for diversity\n",
    "            \"lambda_mult\": 0.6 # Balance factor: 0.6 (0 = maximum diversity, 1 = maximum relevance)\n",
    "        },\n",
    "    }\n",
    "}\n",
    "# Execute the query using MMR search\n",
    "docs = retriever.invoke(\"What is Word2Vec?\", config=config_3)\n",
    "\n",
    "#  Display the search results\n",
    "print(\"\\n🔎 [Search Results - MMR (Diversity and Relevance Balanced)]\")\n",
    "for idx, doc in enumerate(docs):\n",
    "    print(f\"📄 [Document {idx + 1}]\")\n",
    "    print(doc.page_content)\n",
    "    print(\"=\" * 60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbddf405",
   "metadata": {},
   "source": [
    "## Using Separate Query & Passage Embedding Models\n",
    "\n",
    "By default, a retriever uses the **same embedding model** for both queries and documents. However, certain scenarios can benefit from using different models tailored to the specific needs of queries and documents.\n",
    "\n",
    "### Why Use Separate Embedding Models?\n",
    "Using different models for queries and documents can improve retrieval accuracy and search relevance by optimizing each model for its intended purpose:\n",
    "- Query Embedding Model: Fine-tuned for understanding short and concise search queries.\n",
    "- Document (Passage) Embedding Model: Optimized for longer text spans with richer context.\n",
    "  \n",
    "For instance, **Upstage Embeddings** provides the capability to use distinct models for:  \n",
    "- Query Embeddings (```solar-embedding-1-large-query```)  \n",
    "- Document (Passage) Embeddings (```solar-embedding-1-large-passage```)  \n",
    "\n",
    "In such cases, the query is embedded using the query embedding model, while the documents are embedded using the document embedding model. \n",
    "\n",
    "✅ **How to Issue an Upstage API Key**  \n",
    "- Sign Up & Log In: \n",
    "   - Visit [Upstage](https://upstage.ai/) and log in (sign up if you don't have an account).  \n",
    "\n",
    "- Open API Key Page:\n",
    "   - Go to the menu bar, select \"Dashboards\", then navigate to \"API Keys\".\n",
    "\n",
    "- Generate API Key:  \n",
    "   - Click **\"Create new key\"** → Enter name your key (e.g., ```LangChain-Tutorial```) \n",
    "\n",
    "- Copy & Store Safely:  \n",
    "   - Copy the generated key and keep it secure.  \n",
    "\n",
    "<img src=\"./assets/01-vectorstore-retriever-get-upstage-api-key.png\" alt=\"Description\" width=\"1000\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6b51093b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 351, which is longer than the specified 300\n",
      "Created a chunk of size 343, which is longer than the specified 300\n",
      "Created a chunk of size 307, which is longer than the specified 300\n",
      "Created a chunk of size 316, which is longer than the specified 300\n",
      "Created a chunk of size 341, which is longer than the specified 300\n",
      "Created a chunk of size 321, which is longer than the specified 300\n",
      "Created a chunk of size 303, which is longer than the specified 300\n",
      "Created a chunk of size 325, which is longer than the specified 300\n",
      "Created a chunk of size 315, which is longer than the specified 300\n",
      "Created a chunk of size 304, which is longer than the specified 300\n",
      "Created a chunk of size 385, which is longer than the specified 300\n",
      "Created a chunk of size 349, which is longer than the specified 300\n",
      "Created a chunk of size 376, which is longer than the specified 300\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.vectorstores import FAISS\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_upstage import UpstageEmbeddings\n",
    "\n",
    "# ✅ 1. Data Loading and Document Splitting\n",
    "loader = TextLoader(\"./data/01-vectorstore-retriever-appendix-keywords.txt\", encoding=\"utf-8\")\n",
    "documents = loader.load()\n",
    "\n",
    "# Split the loaded documents into text chunks \n",
    "text_splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0)\n",
    "split_docs = text_splitter.split_documents(documents)\n",
    "\n",
    "# ✅ 2. Document Embedding\n",
    "doc_embedder = UpstageEmbeddings(model=\"solar-embedding-1-large-passage\")\n",
    "\n",
    "# ✅ 3. Create a Vector Database\n",
    "db = FAISS.from_documents(split_docs, doc_embedder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "424f69e7",
   "metadata": {},
   "source": [
    "The following example demonstrates the process of generating an Upstage embedding for a query, converting the query sentence into a vector, and conducting a vector similarity search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "aa46e289",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🔎 [Query]: What is an embedding?\n",
      "\n",
      "📄 [Document 1]\n",
      "📖 Document Content: Embedding\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n",
      "📄 [Document 2]\n",
      "📖 Document Content: Definition: Embedding is the process of converting text data, such as words or sentences, into continuous low-dimensional vectors. This allows computers to understand and process text.\n",
      "Example: The word \"apple\" can be represented as a vector like [0.65, -0.23, 0.17].\n",
      "Related Keywords: Natural Language Processing, Vectorization, Deep Learning\n",
      "🗂️ Metadata: {'source': './data/01-vectorstore-retriever-appendix-keywords.txt'}\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "# ✅ 3. Query Embedding and Vector Search\n",
    "query_embedder = UpstageEmbeddings(model=\"solar-embedding-1-large-query\")\n",
    "\n",
    "# Convert the query into a vector using the query embedding model\n",
    "query_vector = query_embedder.embed_query(\"What is an embedding?\")\n",
    "\n",
    "# ✅ 4. Vector Similarity Search (Return Top 2 Documents)\n",
    "results = db.similarity_search_by_vector(query_vector, k=2)\n",
    "\n",
    "# ✅ 5. Display the Search Results\n",
    "print(f\"\\n🔎 [Query]: What is an embedding?\\n\")\n",
    "for idx, doc in enumerate(results):\n",
    "    print(f\"📄 [Document {idx + 1}]\")\n",
    "    print(\"📖 Document Content:\", doc.page_content)\n",
    "    print(\"🗂️ Metadata:\", doc.metadata)\n",
    "    print(\"=\" * 60)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain-opentutorial-bMU5IxA3-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
